home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Magnum One
/
Magnum One (Mid-American Digital) (Disc Manufacturing).iso
/
d20
/
gmd_310.arc
/
GMD.DOC
< prev
next >
Wrap
Text File
|
1991-11-03
|
32KB
|
742 lines
****************************************************************************
** **
** Note: If you are upgrading from 3.0x be sure to read section 9.0. **
** **
****************************************************************************
Grunged Message Detect (GMD) Utility
======================================
Version 3.10
==============
November 3, 1991
==================
Table of Contents
===================
1.0 Warning!
2.0 General Description
3.0 Reasons to Run GMD
4.0 Definition of a "Grunged" Message
5.0 Quick Start
6.0 Operation
6.1 GMD
6.2 GMD-Sort
6.3 GMD-Rpt
7.0 GMD Configuration File
7.1 General Operation Section
7.2 Set Definitions Section
7.3 Default Message Standards Section
7.4 Exception Areas Sections
7.5 What You Should Configure
8.0 Runtime Considerations
8.1 DOS
8.2 OS/2
8.3 Other Operating Systems
9.0 Upgrade Notes
10.0 Licensing Restrictions
11.0 Credits
12.0 History of Changes
1.0 Warning!
=============
GMD is a very powerful program. Used properly, GMD can detect
many types of grunged messages and some types of duplicate
messages.
Used improperly, GMD is capable of deleting valid echo mail,
generating false warning messages, and violating Policy in
general. Thus you should be very careful in setting it up and
in using it. If you haven't read and understood FTS-0001 and
FTS-0004 then you should not attempt to configure or use GMD.
To protect you, me and the rest of the Net, GMD will not delete
an echo mail message without trying to send an information
message to the feed who sent you the packet with the bad
message.
Until you are sure you have your configuration correct, you
should use GMD in the test mode by using the "-t" command line
switch.
2.0 General Description
========================
GMD's main purpose is to detect grunged messages. In the
process of doing this, GMD is also able to identify certain
types of duplicate messages and non-standard messages.
GMD operates directly on incoming packets, as described in
FTS-0001 and FTS-0004.
GMD is very configurable. Indeed, it depends entirely on a
configuration file to tell it what parameters to check the
messages for, and what to do when it finds a "bad" message.
If you have any questions about GMD, or want to discuss any
problems you encountered while running GMD, please use the "GMD"
Echo, if possible. "GMD" is a Backbone echo in Zone 1. It is
the best place to get GMD information and help.
3.0 Reasons to Run GMD
=======================
Grunged messages can cause problems for some mail processing
programs. For example, they can cause certain message
renumbering programs to lockup. Since message renumbering is
usually a part of nightly processing, your system is exposed to
locking up while you are not around. When you discover the
lockup, you are faced with the task of finding the grunged
message, deleting it and then completing the night's processing.
It is not uncommon for this recovery process to last several
hours at an inopportune time.
Deleting grunged messages and duplicate messages spares your
users from seeing them and protects other boards that you feed
echo mail.
Informing the feed who sent you the packet with the bad messages
allows him to correct the problem or to trace it back to it's
source.
GMD is most effective when used close to the source of the
problem. Thus we encourage using GMD at the Net level, in
addition to the Region and Zone levels.
4.0 Definition of a "Grunged" Message
======================================
What is a "grunged" message? This is a fuzzy question so it
gets a fuzzy answer. A grunged message is a message which does
not reasonably conform to FTSC-0001 and FTSC-0004 standards. It
turns out that such strict conformance is not desired because so
many messages fail to strictly conform to the standards (e.g.,
strict conformance to the standard requires two spaces between
the year and time in the date/time stamp field of the message -
it is fairly common to have only one space). Thus strict
conformance would result in detecting messages otherwise
generally considered acceptable.
One of the classic symptoms of a grunged message is non-ASCII
characters throughout the message. Fortunately, this condition
is easy to detect in a message's control fields, especially the
date field which is somewhat rigidly defined. Another form of
grunging, related to a message becoming an unwanted duplicate,
results in additional control fields being appended to the end
of the message. Although not actually a grunged message,
sometimes old messages will get scanned out again, resulting in
duplicates which are too old for conventional dupe checking
techniques to detect. GMD is capable of detecting these sorts
of problem messages.
5.0 Quick Start
================
There is no "quick start" for GMD. We feel that due to the
nature of the program you should read this file in its entirety
before running GMD.
6.0 Operation
==============
6.1 GMD
--------
GMD scans incoming packets. It should be run after unpacking
incoming bundles, perhaps with a program like PolyXArc, but
before importing or tossing the mail. GMD tests each message
for conformance with the criteria specified in the configuration
file. When GMD detects a non-conforming message it can be set
to do nothing or to perform any combination of the following:
1) Generate an "information" net mail message to the feed
who sent you the packet with the bad messages, informing
him of the problem.
2) Delete the non-conforming message from the incoming
packet.
3) Make a "save" copy of the non-conforming message for a
local "bad" area.
GMD only operates on the incoming packets. The original
incoming packets maintain their time/date, even if messages have
been deleted from them. Any messages that GMD generates are put
into new packets, also to be processed by your regular mail
program. GMD does not work with your BBS's message base. Thus
it does not care how your messages are actually stored on your
system, or if they are stored at all (passthrough).
For security reasons, some mail processors will not accept the
messages which GMD generates since they appear to be coming
from that very system itself. In this case GMD should be
configured to generate it's messages as if it is coming from a
Point off your system, rather than your system itself. This
address is set by the System.xxx parameters in the Default
Message Standards Section of the configuration file.
Whatever you set the System.xxx address to, you will need to
make sure that your mail processor will accept routed net mail
(the "information" messages) and echo mail (the "save" messages)
from this address.
GMD puts its FTSC product code (A3) into the header of the
packets as it inspects or generates them. GMD will not process
a packet which has its own product code. Thus GMD will not
process the same incoming packet twice, or any packet which it
generated.
GMD can be called with two optional command line parameters:
-c Specifies a configuration file other than the default
of GMD.Cfg in the current directory, or the directory
where GMD.Exe resides.
-t Specifies "test mode" operation. Used for testing GMD's
setup. Inhibits the "i" and "d" commands. Only the "s"
command is functional. This allows you to test your GMD
setup without disturbing the mail flow or generating
information messages.
GMD should be configured to ignore incoming mail from other
systems which you know are also running GMD. This helps to
avoid sending out redundant "information" messages. When you
set up GMD you should inform all of your echo mail connections
that you are running it, and you should ask them to let you know
if they are, too.
You should then list these addresses in the Ignore_Pkt_From
parameter in the General Operation Section of the configuration
file. You should always list your echo mail feed, even if he
doesn't run GMD. Please see section 7.1 for more information.
We realize that this is not a foolproof scheme, although it
works for most cases. To cover some of the other cases, GMD
also puts a "GMD: xxx" hidden line in any message which causes
it to send out an information message. The "xxx" is the node
who sends out the information message and puts the GMD hidden
line in. Other systems running GMD check for the GMD hidden
line before sending out information messages themselves. The
presence of a GMD hidden line will not stop GMD from deleting a
message if this is called for.
There is a limit to the number of information messages that GMD
will send out due to bad messages in any given packet. This
helps in those cases that every message in a packet is bad.
6.2 GMD-Sort
-------------
GMD generates a statistical database when it operates. The
GMD-Sort program can be used to sort the database, prior to
generating reports from it with the GMD-Rpt program. The
database is sorted and replaced. This does not affect how GMD
itself operates. It merely changes the order of the items in
the report.
To use GMD-Sort, call it with the name of the statistical
database and optionally with one or more of the sort keys. The
sort keys must be separated by at least one space. Each may be
preceded by a "+" (default) or "-", which causes the sort to
happen in ascending or descending order for that key. The
default sort order, if you don't use any of the keys, is "+zone
+net +node +point".
The available sort keys are:
zone Zone number
net Net number
node Node number
point Point number
length Number of messages too long
to Number of errors in To name field
from Number of errors in From name field
subject Number of errors in Subject field
datetime Number of errors in Date/Time field
body Number of errors in body of message
gated Number of errors in Gated Origin line
origin Number of errors in origin line
seenby Number of errors in Seen-By line
path Number of errors in Path line
endseq Number of bad packet endings
info Number of info messages sent
block Number of statistically suppressed messages
delete Number of messages deleted
uninf Number of uninformed deletes
save Number of messages saved
6.3 GMD-Rpt
------------
GMD generates a statistical database when it operates. The
GMD-Rpt program can be used to generate a report from it.
To use GMD-Rpt, call it with the name of the statistical
database. The report will be sent to the console. Redirection
can be used to send the report wherever you desire. For
example, to send it to a file called GMD.Rpt:
GMD-Rpt GMD.Dat >GMD.Rpt
GMD-Rpt can be called with the optional "-s" switch. This will
cause it to only produce the summary.
7.0 GMD Configuration File
===========================
The configuration file contains all the parameters that control
GMD. The GMD distribution file contains a sample configuration
file, called Sample.Cfg. Blank lines and any part of a line
following a ";" are considered comments, and are ignored by GMD.
The configuration file is divided into four major sections:
General General system information
Sets Defines legal zones and character sets
Default Defines the default settings for all echo areas
Area Exceptions for groups of echo areas
Each of these sections starts with appropriate keyword,
immediately followed by a ":". Each section is made up of a
series of parameters. These parameters can be in any order.
A parameter consists of a key word, a "=", and the parameter's
value(s).
For example:
Section_1:
parameter_1 = 12
parameter_2 = 34, "abc", y
parameter_3 = "John Doe"
Any string which contains embedded spaces or other punctuation,
must be quoted. If in doubt - quote it, as it won't hurt.
Any filespec can include a path in addition to the file's name.
Most parameters test for a condition. These parameters are
followed by an "ids" command(s) which tells GMD what to do if
the condition is not met. The "ids" commands are:
i Inform the feed (via net mail) about the problem
d Delete the bad message from the incoming packet
s Save a copy of the bad message
Any combination (including none if no action is desired) of the
"ids" commands may be used, except that if the "d" command is
used, then the "i" command will also be executed, even if you do
not specify it. Some parameters only allow the "is" commands,
and not the "d" command.
If a message fails more than one condition, then the action that
GMD takes is the "or" of the individual conditions' command
settings. For example, if a message fails 3 condition tests,
yet only one of these conditions specifies that a information
message be sent, it will be sent.
GMD permits include files anywhere a token (a string of
characters which GMD recognizes as a unit) can appear. The
syntax is @filespec (or "@filespec"). Include files can be
nested.
Included files can be handy for separate, but similar,
configuration files. They can also be used to input lists of
various sorts (echo areas, names, search strings).
7.1 General Operation Section
------------------------------
This section contains general information about your system and
how GMD should operate.
GMD logs both to the screen and to a file. You can specify the
level of detail you want for each. Each level includes the
information in the lower levels.
LogLevel Char Information Logged
-------- ---- ---------------------
0 ! Errors
1 * Summary
2 + Packets
3 : Bad Messages, brief
4 # Bad Messages, verbose
5 - Good Messages, brief
GMD generates some summary statistics about the numbers and
types of non-conforming messages. This information is in the
logs and can also be sent to you in a net mail message.
The only disk directory that GMD needs to know about is the one
which contains the incoming packets. This is usually your
inbound mail area. GMD does not care what type of message base
your system uses. Any messages that GMD itself generates will
be put in the form of incoming packets. Thus your system treats
them like any other incoming mail. This means that GMD also
does not care what type of outbound area(s) your system uses.
When GMD "saves" a copy of a non-conforming message, it does
this by sending a copy of the message to a "bad" echo area which
you can specify. This allows you to look at the messages later.
GMD is capable of generating a database which contains
statistical information about the number and types of errors
encountered for each node. This database has two uses. It can
be used to limit the number of information messages sent because
of a given node. Also it serves as input to the GMD-Sort and
GMD-Rpt reporting programs. The database is cleared by deleting
the file. A typical use would be to generate a weekly report,
then delete the database.
To prevent your GMD from sending out information messages which
have already been sent out by another node running GMD, you can
instruct GMD to "ignore" packets coming in from certain Nodes.
In fact, GMD will continue to inspect these packets, but will
not generate any information messages or save copies, unless a
delete is also called for, in which case it processes the packet
normally. This allows for the possibility of a message being
grunged after it has been processed by a node running GMD. You
should always list your echo mail feed, even if he doesn't run
GMD.
7.2 Set Definitions Section
----------------------------
This section contains definitions of sets which are used in the
Default and Area Sections. Sets are used to specify character
sets and zone numbers, for example. You can name them whatever
you want to and you can have as many as you need.
Sets make it possible to specify any combination of characters
or numbers which you wish to allow. For example: Just basic,
printing, ASCII characters. Or you might want to allow some
(the alphabetic), but not all, of the hi-bit characters, too.
7.3 Default Message Standards Section
--------------------------------------
This section contains the default settings used for the echo
areas. These set the parameters which determine whether a
message conforms or not. These settings can be overridden for
specific echoes by using the Area Section.
Most of these parameters test conditions, hence use the "ids"
commands. These commands determine what GMD does if a
non-conforming message is found.
Many of the parameters allow you to relax or disable GMD's
conformance checking of various aspects of the message
specifications. At present, there are so many non-conforming
messages that we felt this was necessary until the majority of
the responsible programs were fixed.
Please see section 6.1 for a discussion about how to set the
System.xxx parameters.
GMD uses two template files to build the information messages
that it sends. One template file is used for messages which are
passed and another template file is used for messages which are
deleted. There are a number of macros available which cause GMD
to insert the appropriate information about the offending
message.
These are the macros which can be used in the template files
(note the trailing "."):
&To. The non-conforming message's "to" field
&From. The non-conforming message's "from" field
&Subject. The non-conforming message's "subject" field
&Date. The non-conforming message's "date" field
&Flags. The non-conforming message's flags
&Why. The reason(s) that the message didn't conform
&Area. The echo area that the non-conforming message
was in
&Body. The non-conforming message's body (up to the
origin line)
&Routing. The non-conforming message's control fields
(from the origin line on)
&SysOp. The name specified in System.SysOp
Any other text in the template files is used in the information
message "as is".
The GMD distribution file contains two sample template files.
Sample.Pas is an example of the Info_Message.Pass_File and
Sample.Del is an example of the Info_Message.Delete_File.
Some parameters require sets to represent lists of acceptable
characters. Minimum character sets are OR'ed with the character
sets that you use to prevent leaving out basic characters.
The minimum character set is 32 (space) to 126 (~) for the
following parameters: To.Alphabet, From.Alphabet,
Subject.Alphabet, Gated_Origin.Alphabet and Origin.Alphabet.
In addition to 32 to 126, the minimum character set for the
Body.Alphabet parameter also includes 1 (control A), 9 (tab), 10
(line feed) and 141 (soft carriage return).
7.4 Exception Areas Sections
-----------------------------
These are optional sections which contain exceptions to the
defaults listed in the Default Section. Any setting listed
there can be overridden for specific echoes. An echo mail area
can only be listed in one Exception Area Section.
The only required parameter in an Area Section is "Echoes". It
lists the echo mail areas for which the exceptions which follow
it apply.
Typical uses of Exception Areas include: A Node in more than
one Network who has to cope with different message standards, or
a Node or software developer who wants to do more stringent
testing on certain test areas.
7.5 What You Should Configure
------------------------------
It is very important to configure these parameters to match your
system:
Mail.Inbound
Ignore_Pkt_From
System.Domain
System.Zone
System.Net
System.Node
System.Point
System.SysOp
System.Private_Net (if needed)
These parameters may also be configured to suit your style of
operation:
Log.Screen_Level
Log.Level
Log.File
Log.Error_Stats
Mail.Error_Stats
Mail.Bad_Area
Info_Message.Kill_Sent
Stat.File
Stat.Max_Nr_To_Node
Max_Nr_Per_Packet
Ignore_Pkt_To (if needed)
Ignore_Echo_List (if needed)
Info_Message.Pass_File
Info_Message.Delete_File
Info_Message.Pass_Subject
Info_Message.Delete_Subject
8.0 Runtime Considerations
===========================
GMD returns the following errorlevels:
255 Compiler run time error
3 Configuration file error
2 Bad message deleted, information message sent
1 Bad message passed, information message sent
0 No bad messages or information messages sent
GMD has been assigned FTSC product code "A3".
8.1 DOS
--------
Use the DOS version on 8086/8 CPUs. The OS2 version runs a bit
faster, but it can only be used on 80286, and above, CPUs.
GMD requires about 350K of free memory to run. It is not
overlaid, and does not use EMS or any other type of memory.
GMD running on a 25 MHz 80486, under DesqView, takes about 6
minutes to process 10,000 messages. This is about one full
day's mail for the Zone 1 Backbone as of November 1991.
8.2 OS/2
---------
Use the OS2 version of the programs. They are bound in the
dual mode, thus work in either protected or real mode.
8.3 Other Operating Systems
----------------------------
We would be happy to work with anyone who is interested in
porting GMD to other operating systems. Contact the author.
9.0 Upgrade Notes
==================
There are some changes between version 3.0x and 3.10 which you
need to take into account when upgrading.
There were quite a few changes made to the configuration file.
We recommend that you start over with the enclosed sample
configuration file.
The format of the statistical database has changed. You should
delete the existing database before running the new GMD.
GMD now returns more, and different, errorlevels than it used
to. If you use the errorlevel, you will have adjust for the new
ones.
10.0 Licensing Restrictions
============================
There are some simple restrictions associated with using the
programs. You use these programs at your own risk. The author
does not warrant that the programs work as described herein or
that they are even suitable for any particular purpose. This
documentation is intended to describe how the programs work in
the author's environment. The author does not claim that they
will work the same way in your environment or, if they do work,
that they will continue to work. Furthermore, it is up to you
to decide if these programs are suitable for any particular
purpose on your system.
The author freely releases the executable versions of these
programs to the public domain. The author desires no fees for
the use of these programs and does not support these programs.
These programs compile and link using Borland C++ and Microsoft
C 6.0 with no errors. The source codes for GMD and related
programs are not generally available. However, if you have a
legitimate need, and are willing to sign a non-disclosure
agreement, contact the author.
11.0 Credits
=============
David Troendle, 1:396/5, is the author of GMD and related
programs. As time permits, he is willing to answer questions
about techniques used and how these program generally work.
John Souvestre, 1:396/1, is responsible for the testing and
documenting of GMD and related programs. He also serves as the
interface to the author, when David is busy otherwise.
We would like to thank the many people who helped in one way or
another with GMD's development. Thanks to the following people
who supplied helpful coding ideas or actual code:
Chris Irwin
Jeffrey Nonken
George Peace
Thanks to the beta test team, past and present (*):
Steve Ahola Dean Lachan *
Martin Belcke * Paul Marwick *
Bruce Berna * David Nugent *
Bruce Bodger * Roscoe Primrose *
Bill Bolton Mike Ratledge
Rod Bowman * Bob Rakov *
Larry Carter * Bob Satti *
Tim Carter * Robert Soubie *
Steve Cross * Graham Stair *
Tony Davis * Mort Sternheim *
Fabian Gordon * Roy Timberman *
Jim Greely * John Valentyn *
Miles Hoover * Tony Wagner *
Mark Howard * Adrian Walker *
Dave James * Matt Whelan *
John Johnson * Ken Wilson
Felix Kasza John Woodward *
Thanks to the GMD Echo Moderator, Steve Shapiro.
12.0 History of Changes
========================
Version Date Changes, Modifications, Additions, etc.
------- -------- ---------------------------------------
1.00 12/29/90 Original release
1.01 2/16/91 Maintenance release, a few small fixes
2.00 2/28/91 Added features mainly for Hub usage
- Uses echotoss log
- Uses and updates lastread pointer
- Optional passthrough area processing
- Configurable log detail level
- Returns an errorlevel
3.00 9/12/91 Complete redesign and rewrite!
- Processes incoming packets directly
- Many new parameters
- Parameters now in configuration file
- Ability to send information messages
- Statistical database and reporting
- "Delete" command disabled for now
3.01 9/17/91 Maintenance release
- Fixed Ignore_Pkt_From logic
- Added Seen_By.Node_Sort_OK
3.10 11/03/91 Miscellaneous additions and changes
- "Delete" command enabled, but no
longer allowed in some places
- Fixed Ignore_Pkt_To logic
- Send information messages to feed,
not originator
- Added Max_Nr_Per_Packet
- Added GMD kludge line
- Improved log formats
- Improved report format
- Added more errorlevel returns
- Added Ignore_Echo_List
- Fixed a file handling problem which
caused trouble on certain LANs
- OS/2 version
- End -